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EXECUTIVE SUMMARY 


In October 2009, The New York State Education Department (NYSED), in 
partnership with the New York City Department of Education (NYCDOE), was granted 
funding as part of the Striving Readers Project to address the literacy needs of adolescent 
struggling readers early in middle school. The goal of the project was to implement and 
examine the impact of a one-year comprehensive supplemental literacy intervention that 
was provided to seventh grade students across 11 New York City middle schools. The 
supplemental literacy intervention used in this study was the REWARDS Program 
(REWARDS Secondary-Multisyllabic Word Reading Strategies; REWARDS Plus; 
REWARDS Writing). The REWARDS Program provides comprehensive instruction in 
word analysis, fluency, vocabulary, reading comprehension and writing, and uses 
content-related text and extended discussion of text meaning and interpretation to 
enhance student motivation and engagement in literacy learning. The three components in 
the REWARDS Program were taught in an integrated sequence, by specially trained 
teachers. 


The focus of this report is to describe the characteristics of the study schools, 
teachers, and students to determine the comparability of treatment and control students prior 
to implementation of the REWARDS program. 


Following is a summary of the key demographic findings from the evaluation of the 
treatment and control students for the Striving Readers project: 


e The REWARDS program was implemented in 11 culturally and ethnically diverse 
schools in the New York City area. The schools varied in size, as well as student 
ethnicity. The participating schools had smaller percentages of students earning 
passing scores on the 2009-2010 New York State English Language Arts and 
Mathematics exams when compared to city-wide averages. 


e REWARDS classes at each school were all similar in length and number of 
classes offered. Instead of receiving REWARDS, control students attended 
additional class sessions in the content subjects (1.e., Science or Social Studies) or 
talent/enrichment classes (e.g., Art, Music, Dance, Drama). No direct instruction 
in reading was provided in these classes. 
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The majority of REWARDS teachers were reading teachers or special education 
teachers, prior to beginning the Striving Readers program, and had been teaching 
for at least 5 years. 


Higher attendance rates were observed at professional development trainings by 
the REWARDS teachers compared to their building administrators. Teacher 
attendance in the classroom was calculated based upon self-reported data by the 
teacher. Their absenteeism ranged from 1.6-6% of the school year from October 
through April. 


Direct comparison of treatment and control students indicated equivalence of the 
groups prior to the intervention on both demographic characteristics and the pre- 
test GMRT scores. In most schools, the numbers of treatment and control 
students were equal, or nearly equal, as were their attendance rates. 
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INTRODUCTION 


In October 2009, The New York State Education Department (NYSED), in 
partnership with the New York City Department of Education (NYCDOE), was granted 
funding as part of the Striving Readers Project to address the literacy needs of adolescent 
struggling readers in early middle school. The goal of the project was to implement and 
examine the impact of a one-year comprehensive supplemental literacy intervention that 
was provided to seventh grade students across eleven New York City middle schools. 


The supplemental literacy intervention used in this study was the REWARDS 
Program (Reading Excellence: Word Attack & Rate Development Strategies; 
REWARDS Secondary-Multisyllabic Word Reading Strategies; REWARDS Plus; 
REWARDS Writing). The REWARDS Program provides comprehensive instruction in 
word analysis, fluency, vocabulary, reading comprehension, and writing, and uses 
content-related text and extended discussion of text meaning and interpretation to 
enhance student motivation and engagement in literacy learning. The three components in 
the REWARDS Program were taught in an integrated sequence with careful attention to 
fidelity, by specially trained teachers. Students were randomly assigned to the treatment 
and control groups. Treatment students received the REWARDS program, while control 
students received no additional reading instruction. Groups were monitored throughout 
the year; building, teacher, and student demographic information was gathered from 
multiple sources. 


This report reflects a descriptive analysis of the basic demographic characteristics 
of the schools, teachers, classrooms, and students in the Striving Readers Project. For the 
purposes of this evaluation project, the following objectives will be addressed: 


e Evaluation Objective One: To document the characteristics of the district and 
participating schools. 


e Evaluation Objective Two: To document the characteristics of the REWARDS 
classrooms and Control group activities. 


e Evaluation Objective Three: To document the characteristics of the teachers of 
the REWARDS classes. 


e Evaluation Objective Four: To document the characteristics of the treatment and 
control students. 
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Evaluation Objective One: To document the characteristics of the district and 


participating schools. 


The district and building level demographic variables were gathered from 
multiple sources for examination. The REWARDS program was implemented in 11 
school buildings, located in 4 of the 5 boroughs of New York City. For further 
information on how they were chosen refer to the Random Assignment Report and the 
Evaluation Design Summary. The analysis reflects data from the 2009-2010 school year, 
when sample students were in 6" grade prior to being assigned to treatment or control 
groups. 


I. Building demographics 
Finding: Demographic variables differed across school buildings. 


Table 1 
2009-2010 Building Demographics 


# of Teachers 
Total Working in Building 
Building Enrolled # 6" Graders Building Structure Borough 

A 793 210 55 MS (6-8) Manhattan 
B 555 95 54 IS (6-8) Manhattan 
Cc 828 240 74 JHS (6-8) Manhattan 
D 1152 363 74 JHS (6-8) Bronx 

E 699 211 56 MS _ (6-8) Bronx 

F 2084 611 148 IS (6-8) Queens 

G 1923 595 109 MS _ (6-8) Queens 

H 2038 644 119 JHS (6-8) Queens 

I 625 227 45 (6-8) Queens 

J 887 300 65 IS (6-8) Staten Island 
K 1330 404 88 IS (6-8) Staten Island 


e Shown in Table 1, are the building demographics. The schools included in our 
program were diverse in size and location within the New York City limits. 
Student enrollment ranged from 555 students to 2084 students, and the number of 
teachers in each building ranged accordingly (45-148). Building-wide student: 
teacher ratios ranged from 10:1 (Building B) to 17:1 (Building G). All of the 
schools served grades 6-8, though some are referred to as middle schools, 
intermediate schools, or junior high schools. 
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Table la 
2009-2010 Demographic Information by School Building 
Gender (%) * Ethnicity (%) 
fue’ Male Female ea pees Beet Hawaiian/ Other Caucasian 
Alaska Native Pacific Islander 
A 55.7 44.3 0 6 12 81 1 
B 53.7 46.3 0 48 48 2 
C 54.0 46.0 0 3 96 0 1 
D 47.0 53.0 0 24 64 10 2 
E 54.3 45.7 0 27 70 2 1 
F 54.5 45.5 0 6 81 12 2 
G 52.6 47.4 0 8 35 54 2 
H 50.2 49.8 0 7 61 28 4 
I 54.1 45.9 1 20 56 15 8 
J 52.0 48.0 1 41 34 11 13 
K 44.4 55.6 0 39 34 5 21 


Note: * estimated based upon number of students completing state tests 


Il. 


Presented in Table la are the student demographic characteristics in the 
participating schools. The majority of the student population in the participating 
buildings was male, with the exception of schools D and K. 


Generally, the ethnicity of students differed across the school buildings. The 
majority of students were Hispanic/Latino in 6 of the 11 schools (Buildings C, D, 
E, F, H, & I). In the two schools (Buildings A & G), the majority of students 
were Asian/Native Hawaiian/Other Pacific Islander. Variability, from 3-48%, 
was noted in the proportion of students identified as African American across the 
buildings. Caucasian students comprised at most 21% of the school population 
(Building K), though in most schools they were represented to a limited extent (1- 
4%). American Indian or Alaska Natives made up only 1% of the population in 
two schools. For further demographic information (i.e., special education 
students, free and reduced lunch, English Language Learners) refer to the Fidelity 
of Classroom Implementation and Implementation Fidelity Score Report (pp.12- 
13). 


State test scores by district/borough/building 


The following data represent state test scores in English Language Arts and 


Mathematics for 6" grade students in the 2009-2010 school year (pre-intervention). 
Scores on these tests ranged from 1-4, with scores of 3 and above as passing (i1.e., 1: Not 
Meeting Learning Standards; 2: Partially Meeting Learning Standards; 3: Meeting 
Learning Standards; 4: Meeting Learning Standards with Distinction). Students were 
chosen from the pool of students who scored at level 2 on the New York State English 
Language Arts (NYS-ELA) exam (for further information, refer to the Random 
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Assignment Report). Below are city-wide data by borough, ethnicity, and other 
demographic variables, as well as the data for the specific schools in the Striving Readers 
project. 


Finding: Student scores on the state English Language Arts exam varied across different 
boroughs and student demographics. 


Table 2 
2009-2010 Sixth Grade NYS-ELA Performance 
Percent of Students Performing at Level: 
Category 
1 2 3 4 
City-Wide 17.2 42.6 36.6 3.6 
ses 12.7 42.8 40.5 4.0 
: Proficient 
English : 
Proficiency Boe) 
Language 53.4 41.3 5.2 0.1 
Learners 
ochre 10.4 42.5 42.8 43 
Special Education ; , , ; 
cea ae 47.2 43.4 9.2 0.2 
Education 
Gendst Female 14.4 41.5 39.6 4.5 
Male 19.9 43.8 33.6 2. 
Asian 8.0 28.2 54.2 9.7 
hanes Black 20.4 49.4 28.7 1.6 
Hispanic 21.8 47.2 29.7 1.3 
White 7.0 30.0 55.0 8.0 
Queens 12.6 37.9 43.8 5.7 
Manhattan 16.4 42.1 36.7 4.8 
Borough Bronx 23.9 48.6 26.5 0.9 
Brooklyn 17.9 43.7 35.4 2.8 
Staten Island 13.5 38.9 43.2 4.4 
A 31.0 40.0 28.0 1.0 
B 47.0 43.0 10.0 0.0 
C 28.0 47.0 25.0 0.0 
D 13.0 45.0 39.0 3.0 
E 39.0 50.0 11.0 0.0 
School F 25.0 47.0 25.0 3.0 
G 10.0 40.0 45.0 5.0 
H 12.0 38.0 45.0 5.0 
I 26.0 47.0 27.0 0.0 
J 33.0 49.0 17.0 1.0 
K 20.0 44.0 33.0 3.0 


e Illustrated in Table 2 are 6" grade 2010 New York State English Language Arts 
(NYS-ELA) scores for students throughout New York City, by borough and 
participating school. City-wide, 40.2% of students met or exceeded state 
standards (i.e., performance levels 3 and 4). Furthermore, the percentage of 
English Proficient students (44.5%) scoring at level 3 or 4 was higher than the 
percentage of English Language Learners (5.3%). Similarly, a higher proportion 
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of general education students (47.1%) scored at passing levels compared to the 
proportion of students enrolled in special education (9.4%). 


e Meeting or exceeding state standards varied by gender. Specifically, the 
percentage of females (44%) meeting or exceeding standards was higher than the 
percentage of males (36.3%). 


e Differences in test performance were also noted by ethnicity. Specifically, twice 
as many students identified as Asian (63.9%) and White (63%) scored at level 3 
or 4 compared to students identified as Black (30.3%) and Hispanic/Latino (31%). 


e Passing rates were noted to vary across the different boroughs. Queens and Staten 
Island had the highest percentage of students scoring at levels 3 and 4 (49.5% and 
47.6%), and the Bronx was the district with the smallest percentage of students 
earning passing scores (27.4%). 


e When compared to city-wide ELA results, 8 of the schools in this study had fewer 
students meeting or exceeding state standards. In all schools, except schools B, G 
and H, the largest percentage of students performed at level 2 on the NYS-ELA 
exam. Moreover, in all the schools in our study, 5% or less of the students scored 
at level 4. 


Finding: Student test scores on the state math exam varied by borough and student 
characteristics. 


Table 3 
2009-2010 Sixth Grade Math Performance 


Percent of Students Performing at Level: 
Category 
1 2 3 4 
City-Wide 11.6 35.4 29.7 23.3 
Pouch 9.1 33.5 31.4 26.0 
. Proficient 
English ; 
Proficiency Enetsh 
Language 27.8 48.1 18.0 6.1 
Learners 
San 6.6 32.7 32.9 27.8 
Special Education ; : ; ; 
Eueene? Special 33.8 47.3 15.3 3.6 
Education 
cide Female 10.2 35.2 30.3 24.4 
Male 13.0 35.6 29.1 22.3 
Asian 3.0 15.0 29.8 52.2 
Ethnicit Black 15.9 44.5 2D 12.1 
y Hispanic 13.9 41.2 29.6 15.2 
White 4.6 20.8 34.3 40.3 
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Cc Percent of Students Performing at Level: 
ategory 
1 2) 3 4 
Queens 8.1 30.0 31.4 30.5 
Manhattan 11.3 35.0 28.7 25.0 
Borough Bronx 16.7 42.7 27.6 13.0 
Brooklyn 11.8 36.1 29.3 22.8 
Staten Island 8.8 31.6 33.1 26.5 
A 8.0 34.0 31.0 27.0 
B 26.0 60.0 10.0 4.0 
C 15.0 45.0 30.0 10.0 
D 9.0 30.0 37.0 24.0 
E 29.0 51.0 17.0 3.0 
F 14.0 43.0 28.0 15.0 
G 5.0 29.0 38.0 28.0 
H 7.0 30.0 32.0 31.0 
I 15.0 50.0 26.0 9.0 
J 24.0 47.0 21.0 8.0 
K 15.0 43.0 26.0 16.0 


Illustrated in Table 3 are 6" grade 2010 New York State Math (NYS-Math) scores 
for students across New York City, in each borough, as well as each school. City- 
wide, just over 50% of the sixth graders met or exceeded state standards. Similar 
rates of passing were observed for males and females (51.4% and 54.7%). 


Student passing rates varied by student language proficiency status, and by 
educational placement. More than two times as many English proficient students 
(57.4%) scored at level 3 or 4 compared to students identified as English 
Language Learners (24.1%). A higher percentage of students in general 
education (60.7%) passed the math test than students receiving special education 
services (18.9%). 


Math test performance differed by student ethnicity. Greater than 75% of 
students classified as Asian and White scored at level 3 or 4, and less than 50% of 
students classified as Black and Hispanic scored at these levels. 


Meeting or exceeding state standards in math varied across the different boroughs. 
The majority of students in Queens, Manhattan, Brooklyn, and Staten Island 
scored in the passing range on the exam. 


When compared to city-wide statistics, 4 of the participating schools had a similar 
or higher percentage of students performing at level 3 or 4. 
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Evaluation Objective Two: To document the characteristics of the REWARDS 


classrooms and Control group activities. 


Classroom demographics for treatment and control students were examined, 
including length and number of REWARDS classes, student: teacher ratios, class sizes, 
and control student class assignment. A number of different sources were reviewed to 
gather the necessary data (e.g., classroom observations, attendance records, school 
websites, teacher and student reports). These data were collected during the 2010-2011 
school year during the implementation of the REWARDS program, when sample 
students were in seventh grade. 


I. REWARDS Classroom Data 


Finding: Class period length was similar across the different schools. The number of 
classes offered varied across the buildings. 


Table 4 
Length and Intensity of REWARDS Program Provided 
Average Length of Total Number of 
REWARDS Classes # REWARDS REWARDS Classes 
Building (minutes) Sections Offered 
A 45 1 134™ 
B* 45 1 n/d 
C * 46 1 136° 
D * 42 2 1287 
E 42 1 119™ 
Fe 43 5 130" 
G 42 1 129° 
H * Al 1 n/d 
I 41 1 138° 
J* Al 3 135™ 
K 44 1 1347 
MEAN 43 


Note: * attendance data incomplete for this school 

n/d no data are available at this time 

” attendance data calculated through the end of May, 2011 

“ attendance data calculated through the end of April, 2011 

’ attendance data calculated through the end of June, 2011 

shaded area: average was not calculated due to lack of data received 


e Shown in Table 4, are data reflecting the length and intensity at which the 
REWARDS program was provided. In general, the length of REWARDS classes 
was similar across the schools (41-46 minutes/class). Each building offered the 
REWARDS class 5 times a week, though not necessarily every day (e.g., two 
sessions on Wednesday, none on Tuesday). When the REWARDS class was 
scheduled during the school day varied across buildings as well (e.g., during 
optional extended school day hours). The number of REWARDS sections offered 
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differed (1-5 sections), largely due to the number of students in the building, and 
other administrative decisions. 


The total number of REWARDS classes offered was calculated based upon 
teacher reported attendance data. The actual number of days of instruction for the 
REWARDS program appears to have varied across the buildings (119-139 days); 
however, this conclusion is tentative because not all attendance data were 
received by the evaluation team. For further information, refer to the Fidelity of 
Classroom Implementation and Implementation Fidelity Score Report (pp. 15-16). 


Finding: Student:Teacher ratios in the REWARDS classes varied by school building. 


Table 5 


Student:Teacher Ratios by School Building 


Student: Teacher 
Ratio 


A 10:1 
21:1* 
15:1 
19:1 
19:1 

11.2:1 
16:1 
19:1 
12:1* 

14.7:1 
12:1 

Note: * Some students in class are not part of 

treatment group 


Building 


Al alH LQ} am olalw 


Illustrated in Table 5 are student:teacher ratio data from REWARDS classes. 
Ratios ranged from 10:1 to 21:1; all schools with the exception of school B 
remained under the specified limit of 20:1 set by the REWARDS program 
developer. Two buildings, schools B & I, had students in the REWARDS classes 
that were not part of the study, but received the intervention due to administrative 
decisions (data from these students were NOT included in any data analyses). 
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Finding: REWARDS class sizes varied across school buildings. 


Table 6 
Class Sizes by School Building 
Class Size 
Building Starting Class | Ending Class 
Size Size 
(average) (average) 
A 10 10 
B 4 4 
C 15 15 
D 20 19 
E 20 19 
F 14.6 11.2 
G 20 16 
H 20 19 
I 8 7 
J 16 14.7 
K 18 12 


e Shown in Table 6 are REWARDS class size data, calculated from class lists. The 
class sizes range from 4-20 treatment students. Most schools lost students to 
attrition throughout the school year, with the exception of schools A, B and C. 
The overall attrition rate was low (n=14). For more information on attrition, refer 
to the Random Assignment Report (p.14). 


II. Control Student Classroom Data 


An important demographic variable is the activities of the students in the control 
group in place of intervention services. These students were to receive only the typical 
reading/language arts instruction allotted to pupils in grade 7. To determine control 
student class assignment, requests were made to building administrators to obtain copies 
of all student schedules in the Striving Readers project. 


Finding: Students in the control group were assigned to a variety of non-language arts 


classes across the schools in the project. 


Table 7 


Control Student Class Assignment by School 


Type of Class 
Building és 2 
Content Talent/Elective Gym Enrichment Not Reported 
Y Y 
B Y 
Cc Y Y 
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Building Type of Class | 
Content Talent/Elective Gym Enrichment Not Reported 
x 
E Y Y Y 
F Y Y 
G Y 
< x 
: Y 
x 
. x 


Content=extra Science or Social Studies Class; Talent/elective=e.g., Art, Music, Dance, Drama; 
Enrichment=e.g., Scientific Calculator, School Newspaper, Digital Portfolio 


e Information regarding class assignments for control group students was provided by 


the administrators from seven of the schools (see Table 7). 


Most often, control 


students attended additional class sessions in the content subjects (i.e., Science or 
Social Studies) or talent/enrichment classes (e.g., Art, Music, Dance, Drama). No 
direct instruction in reading was provided in these classes. 
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Evaluation Objective Three: To document the characteristics of the teachers of the 


REWARDS classes. 


A number of teacher demographic variables, based on information gathered from 
multiple sources, were examined (i.e., number of years teaching, prior positions, and 
attendance in classes and at professional development). The sources included post- 
professional development teacher surveys, professional development attendance sheets, 
and self-reported classroom attendance data. Results of the REWARDS classroom 
attendance data should be interpreted with caution as they were recorded by the teachers 
themselves. 


Finding: The background of teachers implementing the REWARDS program varied. 


Table 8 
Teacher Demographics by School Building 
# Years 
Position Prior # Years Teaching at 
Building Gender to REWARDS Teaching School 
A F SE 7 5 
B F Other 20 10 
Cc F RT 8 1 
D F N/A 6 6 
E F Other 7 7 
F F RT 8 8 
G F RT 2 2 
H M SE ) 4 
I F RT 8 6 
J F RT 8 7 
K F Other 5 5 
AVERAGE 7.64 5.55 


Note: SE — Special Education Teacher; RT — Reading Teacher; Other — Data Specialist, Team 
Leader or Health teacher; N/A — that information was not provided on survey 


e Shown in Table 8, are teacher backgrounds and experience data prior to the 
REWARDS program. The majority of teachers had backgrounds in reading or 
special education prior to beginning the REWARDS program. Other positions 
included team leader, health teacher and data specialist. Most of these teachers 
had been teaching for at least 5 years, with the totals ranging from 2-20 years 
(mean=7.6 years). Similarly, the total number of years teachers had been in their 
building ranged from 1-10 years, with the average of 5.6 years. 


Finding: Teacher and administrator attendance at Professional Development differed 
across school buildings, as did teachers’ classroom attendance. 
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Table 9 
Teacher & Administrator Classroom and Professional Development Attendance 
Data 
% Days absent from % Teacher % Administrator 
Building REWARDS Attend PD* Attend PD 
A 6.0 90.9 100.0 
B N/A 63.6 33.3 
C 5.1 90.9 66.6 
D 2.3 100.0 66.6 
E 5.9 81.8 66.6 
F 3.6 90.9 100.0 
G 1.6 72.7 66.6 
H N/A 81.8 33.3 
I 3.6 90.9 66.6 
J 3.0 100.0 33.3 
K 3.7 90.9 33.3 
AVERAGE 3.87 86.8 60.6 


Note: N/A indicates not enough attendance records were received to make a reliable estimate; * attendance 
data are incomplete as September 2010 attendance was not received; thus scores represent 11 of 12 trainings. 


e Shown in Table 9 are classroom and professional development training attendance 
rates. Overall, at professional development, teachers had better attendance than 
administrators. For further details see Professional Development Fidelity report 


(pp. 17-18). 


e In the classroom, the percent of days the teacher was absent was calculated using 
attendance data reported by each teacher. The percent of days absent ranged from 
1.6-6% of the school year from October through April. Evaluators observed 
teachers’ notations indicating reasons for some of the absences, such as grading 
state exams or illness. It is important to note that teacher attendance was 
determined through the examination of archival student attendance records kept 
by participating teachers. Most teachers marked their own attendance along with 
the student attendance. Some, however, did not provide any data. Thus, the 
teacher absence data should be interpreted with caution. 
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Evaluation Objective Four: To document the characteristics of the treatment and 


control students. 


Data regarding treatment and control student demographics were gathered from 
multiple sources. Basic demographic information was analyzed from original student data 
files used in the initial random assignment of students to treatment and control groups. 
Building attendance data were collected from schools on a monthly basis. The final data 
were received in May 2011, so percentages are approximate. To determine the 
equivalence between the treatment and control groups, baseline demographics and Gates- 
MacGinitie Reading Tests (GMRT) scores were compared. 


Finding: The percentage of treatment and control students in each building was 
relatively stable. 


Table 10 
Treatment and Control Students by Building 
Group 
Total # Students in Treatment Control 
Building Study % (n) % (n) 
A 20 50.0 (10) 50.0 (10) 
B 8 50.0 (4) 50.0 (4) 
C 30 50.0 (15) 50.0 (15) 
D 78 48.7 (38) 51.3 (40) 
E 36 50.0 (18) 50.0 (18) 
F 128 44.5 (57) 55.5 (71) 
G 35 45.7 (16) 54.3 (19) 
H 40 50.0 (20) 50.0 (20) 
I 15 53.3 (8) 46.7 (7) 
J 91 49.5 (45) 50.5 (46) 
K 26 46.2 (12) 53.8 (14) 
TOTAL 507 47.9 (243) 52.1 (264) 


e Illustrated in Table 10 are data about the number of treatment and control students 
in each building in the final sample after attrition. The total number of students in 
the study ranged from 8-128 across buildings. In most schools, the number of 
treatment and control students was equal, or nearly equal. The school with the 
largest difference was School F. For information about group attrition rates after 
Intent to Treat see the Random Assignment Report (pp.14, Table 5). 
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Finding: Student demographic characteristics were similar across the groups. 


Table 11 
Treatment and Control Student Demographics 


Demographic Treatment Students Control Students 
Variable Group % (n) % (n) x (dfy** 
Male 47.7 (116) 54.9 (145) 2.62 (1) 
pane Female 52.3 (127) 45.1 (119) 
Asian 11.5 (28) 12.9 (34) 2.58 (4) 
wu Black 23.0 (56) 20.1 (53) 
ok 
ee Hispanic 63.0 (153) 63.6 (168) 
White 25 (6) OF 
: : General Education 95.9 (233) 95.5 (252) 0.06 (1) 
PR USO ll aS eetial Eaucation 4.1 (10) 4.5 (12) 
7 Yes 90.9 (221) 88.3 (233) 0.98 (1) 
English Proficient No 9.1 (22) 17 Ql) 
Free and Reduced No 5.3 (13) 5.3 (14) 0.00 (1) 
Lunch Yes 94.7 (230) 94.7 (250) 
Note: N=507 


* indicates that two students were in the “Other” category, but removed because there was no 
comparable city-wide testing data as reported in Tables 2 & 3. 
** qll p > .05-not significant 


Illustrated in Table 11 are treatment and control student demographic variables. 
Across all characteristics the groups were similar. The entire sample consisted of 
more males than females. In terms of ethnicity, the majority of the students 
represented diverse backgrounds, with the largest subgroup of students identified 
as Hispanic (63%). Less than 3% of the students were classified as white. 
Approximately 95% of the sample were eligible for free/reduced lunch (an 
indicator of lower socio-economic status). The vast majority of the sample were 
general education students (~96%). One of the exclusions for selecting the 
sample was that students with an IEP for reading were not eligible, thus the low 
percentage of students in the sample receiving special education. Finally, over 
88% of the students were classified as proficient in English. 


Pearson’s Chi-Square was used to test the hypotheses that students in the 
treatment and control groups were similar for each demographic characteristic. 
No significant differences were noted between the groups; the treatment and 
control students were similar on every characteristic examined. 
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Finding: Treatment and control student school attendance rates were similar. 


Table 12 
Treatment and Control Student Building Attendance 
% of Days in Attendance 
Building Treatment Students Control Students Difference 
Mean (SD) Mean (SD) 
A 98.23 (1.38) 98.28 (1.47) 0.16 
B 92.92 (4.36) 94.44 (4.82) -1.52 
C 97.67 (2.55) 97.04 (3.22) 0.63 
D 94.39 (7.02) 94.01 (5.67) 0.38 
E 93.07 (7.93) 94.17 (4.34) -0.95 
F 93.73 (8.24) 95.04 (5.29) -1.11 
G 92.81 (5.75) 96.20 (3.22) -3.36 
H 92.58 (5.37) 95.03 (5.71) -2.45 
I 96.88 (2.91) 96.27 (3.94) 0.61 
J 93.37 (6.85) 92.17 (7.82) 1.20 
K 93.80 (6.73) 93.53 (6.18) -0.13 
AVERAGE 94.06 (6.80) 94.57 (5.69) -0.51 


Note: Attendance rates were calculated using data received as of May 25, 2011 


e The evaluation team examined the building attendance data for treatment and 
control students. The findings can be seen in Table 12. Overall, these data 
suggest that the sample students had a high rate of school attendance. 


e More variability in attendance rates, as suggested by the larger standard 
deviations, was observed in schools D, E, & F for the treatment students 
compared to their control student counterparts. 


Finding: Pre-test GMRT scores were comparable across the groups. 


Table 13 
Sixth Grade Treatment and Control Student GMRT Pre-Test Score Data 


Test/Score Type Group N M SD t df 
Gates Mac-Ginitie Reading Test REWARDS 257 39.12 12.67 0.98 | 524 
(GMRT)- Normal Curve Equivalent Control 269 38.06 12.39 


Note: p > .05 — not significant 


e Displayed in Table 13 are the results of the independent sample t-test comparing 
the group means on the pre-test GMRT. On a nationally-normed reading test 
(GMRT), the REWARDS and control groups earned comparable scores prior to 
entering the study; no significant differences were identified between the groups. 
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SUMMARY 


The New York State Education Department, in partnership with the New York 
City Department of Education, developed a Striving Readers project to address the 
literacy needs of early adolescent struggling readers in middle school. The goal of the 
project was to implement and examine the impact of a one-year comprehensive 
supplemental literacy intervention offered in New York City middle schools. 


This report reflects comparability of the treatment and control students prior to 
program implementation. Students were compared on the individual level as well as on 
building, teacher, and classroom levels. In review, 


e The 11 schools included in the study were drawn from 4 of the 5 boroughs of 
New York City, and were relatively homogeneous. Most were primarily male, 
and Hispanic/Latino. Furthermore, the majority of the students represented 
diverse backgrounds. The schools served grades 6-8, though ranged in size as 
well as number of teachers. The sample students were chosen from the pool who 
had earned scores of 2 on the 2009-2010 New York State English Language Arts 
(NYS-ELA) exam. The 11 participating schools had smaller percentages of 
students receiving passing scores on the exam. Similarly, most of the schools had 
fewer students earning a passing score on the 2009-2010 New York State Math 
(NYS-Math) exam. 


e Describing the classroom experiences of treatment and control students is crucial 
to the evaluating the integrity of the study design. REWARDS classes were all 
similar in length and number of classes offered. Instead of receiving REWARDS 
control students attended additional class sessions in the content subjects (i.e., 
Science or Social Studies) or talent/enrichment classes (e.g., Art, Music, Dance, 
Drama). No direct instruction in reading was provided in these classes. 


e Teachers who taught the REWARDS classes came from diverse backgrounds and 
had different levels of experience prior to teaching REWARDS. The majority of 
teachers were reading teachers or special education teachers, prior to beginning 
the REWARDS program, and had been teaching for at least 5 years. Teachers 
and administrators received additional training specific to REWARDS. Higher 
attendance rates at professional development trainings by the REWARDS 
teachers compared to their building administrators were observed. Teacher 
attendance in the classroom was calculated based upon self-reported data by the 
teacher. Their absenteeism ranged from 1.6-6% of the school year from October 
through April. It is important to note that teacher attendance was determined 
through the examination of archival student attendance records kept by 
participating teachers. Thus, the teacher absence data should be interpreted with 
caution. 


e Direct comparison of treatment and control students indicated equivalence of the 
groups prior to the intervention on both demographic characteristics and the pre- 
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test GMRT scores. The total number of students in the study ranged across 
buildings from 8-128. In most schools, the numbers of treatment and control 
students were equal, or nearly equal. School attendance rates were, on average, 
nearly equal for treatment and control students, with percentages ranging from 92- 
98% across schools. 
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